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PEER  RATINGS:  SCORING  STRATEGY  DEVELOPMENT  AND  RELIABILITY 
DEMONSTRATION  ON  AIR  FORCE  BASIC  TRAINEES 


I.  INTRODUCTION 

Personnel  selection  research  in  the  U.S.  military  has  typically 
employed  paper-and-pencil  tests  of  various  types  to  predict  perform¬ 
ance  in  training  and  on  the  job.  However,  another  strategy  for 
predicting  training  and  job  performance  In  the  military  has  been  to 
gather  peer  ratings  of  pote'ntial  for  effectiveness.  Peer  ratings  have 
enjoyed  considerable  success  In  predicting  a  variety  of  effectiveness 
criteria  in  the  military.  Within  the  Air  Force,  peer  ratings  have 
correlated  successfully  with  pilot  training  success  (Flyer  &  Bigbee, 
1955),  instructor  pilot  job  performance  (Swanson  &  Johnson,  1975), 
Officer  Training  School  graduate  performance  (Tupes,  1957),  and 
performance  as  cadets,  officer  candidates,  and  officers  (Tupes  & 
Kaplan,  1961).  Annotations  of  these  studies  and  the  considerable  peer 
rating  research  conducted  in  the  Army,  Navy,  and  Marine  Corps  can  be 
found  in  Lammlein  and  Borman  (1979). 

The  objective  of  the  present  study  was  to  evaluate  the  psychometric 
characteristics  of  peer  ratings  among  Air  Force  basic  trainees.  If 
the  psychometric  properties  of  these  peer  ratings  are  satisfactory, 
then  future  research  should  access  the  utility  of  peer  evaluations,  by 
themselves  and  in  combination  with  test  scores,  for  predicting  success 
in  advanced  training  and  subsequent  job  performance. 


Special  Problems  in  Scoring  Nominations 

Raters  in  the  present  study  were  asked  to  identify  the  five  best  or 
most  effective  trainees  in  their  flight  (Ns  3  29-58)  on  each  of 
several  categories  (e.g..  Try  Hard,  Cooperative).  They  were  also 
asked  to  identify  the  five  worst  or  least  effective  trainees  in  their 
flight  on  each  of  these  categories.  The  main  problem  this  kind  of 
data  presents  is  how  to  score  nominations  when  different  groups 
contain  different  numbers  of  persons  and  scores  must  be  compared 
across  groups.  Kane  and  Lawler  (1978)  and  Willingham  (1959a)  present 
detailed  descriptions  of  this  problem. 

Briefly  the  difficulties  seem  primarily  from  two  sources.  The  most 
obvious  problem  is  that  members  of  large  groups  have  a  qreater 
potential  of  receiving  many  nominations  (negative  or  positive)  than  do 
members  of  smaller  groups.  This  creates  larger  variances  In  raw 
scores  for  larger  groups  and  results  in  non-comparable  nomination  raw 
scores  across  different-sized  groups.  The  difficulty  has  typically 
been  addressed  by  dividing  the  sum  of  the  nominations  for  each  group 
member  by  the  total  number  of  nominators  in  the  group;  however, 
Willingham  (1959a)  argued  that  this  correction  does  not  in  itself 
equate  scores  across  groups.  As  evidence,  Willingham  demonstrated 
that  the  adjusted  scores  correlate  highly  (and  negatively  with  the 
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size  of  the  groups,  and  therefore  this  adjustment  provides  an  over¬ 
correction,  (i.e.,  corrected  scores  in  larger  groups  now  tend  to  have 
smaller  variances).  Willingham  offered  another  correction  formula 
that  successfully  equates  the  variances  of  adjusted  scores  for 
different-sized  groups,  and  thus  aids  in  creating  comparable  scores 
across  groups. 

Willingham's  formula  does  not  address  the  problem  of  differing 
kurtosis  (peakedness  of  the  distribution)  for  different-sized  groups. 
Larger  groups  tend  to  have  more  peaked  distributions  than  do  smaller 
groups  because  a  large  proportion  of  the  members  of  large  groups 
typically  receive  zero  or  a  small  number  of  nominations.  This  too 
creates  inequities  in  scoring  nominations  for  groups  of  different 
sizes,  and  Willingham  (1959a)  suggested  that  similar  distributions 
across  groups  could  be  obtained  by  asking  group  members  to  provide 
different  numbers  of  nominations  according  to  their  own  group's  size. 
He  presented  a  formula  to  compute  the  number  of  nominations  required 
for  differing  group  sizes. 

For  purposes  of  the  present  project,  it  was  not  feasible  to  employ 
these  suggestions  directly.  First,  Willingham's  correction  formula 
assumes  that  persons  can  nominate  themselves,  while  the  Instructions 
in  this  peer  nomination  administration  asked  that  persons  not  nominate 
themselves.  Also,  his  formula  requires  that  each  group  member 
nominated  provides  nominations,  and  this  requirement  was  not  met  in 
the  present  stuay.l  Willingham's  suggestion  for  handling  differing 
levels  of  kurtosis  could  not  be  employed,  either,  because  data  had 
already  been  collected  on  a  form  requiring  five  positive  and  five 
negative  nominations  for  all  groups.  Accordingly,  for  the  present 
study  a  norming  strategy  was  devised  in  which  flights  with  similar 
numbers  of  nominees  were  pooled  together,  and  then  standardized 
nomination  scores  were  developed  for  each  grouping.  These  procedures 
are  discussed  in  the  Method  section. 


II.  METHOD 


Subjects 

Subjects  in  the  present  research  were  more  than  24,000  students 
assigned  to  Air  Force  Basic  Training  at  Lackland  AFB,  Texas.  The  mean 
age  of  the  trainee  sample  was  20.29,  with  a  standard  deviation  of  2.17. 


^Typically,  flights  (student  groups)  contained  more  nominees 
than  nominators  because  trainees  were  instructed  to  nominate  anyone  in 
their  flight,  including  those  ./ho  for  various  reasons  were  absent  from 
the  rating  schedule. 
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Data  Collection 


The  peer  nominations  were  gathered  by  staff  members  of  the  basic 
training  course.  In  general,  data  collection  was  accomplished  for 
each  flight  separately.  Student  raters  were  given  a  roster  of  their 
flight  members  and,  as  mentioned,  were  instructed  to  nominate  the  five 
best  and  five  worst  trainees  on  each  of  eight  rating  dimensions.  They 
were  told  not  to  nominate  themselves. 


Method  for  Scoring  the  Nominations 

A  special  norming  strategy  was  developed  for  adjusting  raw  nomination 
scores  so  that  they  would  be  comparable  across  the  different  flights. 
Specifically,  the  strategy  combined  empirical  norming  and  a  procedure 
to  equate  the  expected  variances  of  adjusted  scores  for  different 
flights.  Flights  with  approximately  equal  numbers  of  nominees  were 
grouped  together  (the  number  of  nominators  varied  to  a  degree  across 
these  groups^),  and  this  brought  together  flights  with  very  similar 
levels  of  kurtosis,  because  the  expected  kurtosis  varies  according  to 
number  of  nominees.  Unfortunately,  the  expected  variances  of  the 
distributions  of  these  different  groups  were  highly  dependent  on  the 
number  of  nominators  in  the  groups  and,  therefore,  an  adjustment  In 
peer  nomination  scores  was  required  to  equate  the  expected  variances. 

To  deal  with  the  problem,  a  formula  was  developed  to  adjust  nomination 
scores,  resulting  in  vhe  expected  variances  being  equated  for  flights 
with  differing  number'  of  nominators.  Then,  these  adjusted  nomination 
scores  (for  each  grouping  of  flights  separately)  were  aggregated  in  a 
single  frequency  distr :bution,  and  a  percentile  score  was  assigned  to 
each  score.  Finally,  standard  scores  were  developed  for  each  adjusted 
nomination  score  based  on  the  corresponding  percentile  of  a  normal 
distribution. 

In  general,  these  adjustments  in  raw  scores  allowed  statistically 
justified  comparisons  to  be  made  of  peer  nomination  scores  from 
different  flights.  Importantly,  the  distributions  of  adjusted  scores 
for  individual  flights  eliminated  characteristics  of  the  raw  score 
distributions  known  to  be  artifacts  of  the  peer  nomination  measurement 
process.  On  the  other  hand,  these  "new"  distributions  did  retain  the 
characteristics  that  -t fleet  valid  variations  across  the  different 
groups  of  nominees.  Fee  example,  with  this  scoring  system  if  one 
flight  had  two  or  three  trainees  consistently  nominated  positively  on 
a  rating  category,  and  a  second  flight  in  the  same  grouping  had 
positive  nominations  on  this  category  spread  more  evenly  across  flight 


^Many  flights  containe-d  more  nominees  than  nominators. 
However,  for  some  flights  tnere  were  many  more  nominees  than 
nominators  and  in  other  flights  almost  every  nominee  served  as  a 
nominator.  Thus,  when  flights  were  grouped  according  to  number  of 
nominees,  the  number  of  nominators  varied,  considerably  in  some  cases. 


members,  then  those  two  or  three  in  the  first  flight  would  tend  to 
receive  higher  standard  scores  than  the  top  two  or  three  in  the  second 
flight.  In  sum,  the  scoring  procedures  outlined  above  appeared  to 
meet  the  critical  requirement  of  allowing  justified  comparisons 
between  trainees  across  flights,  and  therefore  standardized  scores 
were  computed  for  each  trainee  on  each  rating  category  and  these 
scores  were  used  in  correlational  analyses. 


Method  for  Evaluating  the  Interrater  Reliability  of  the  Nominations 

The  approach  to  obtaining  interrater  reliabilities  involved  computing 
special  intraclass  correlation  coefficients  for  each  rating  category 
within  flights  and  then  examining  the  distributions  of  those 
reliabilities  for  each  category.  Derivation  of  a  more  general 
formulation  for  the  intraclass  correlation  was  required  because  no 
existing  formula  was  appropriate  for  this  data  set.  In  particular, 
the  data  suffered  from  the  "diagonal  problem,"  i.e.,  trainees  in  the 
sample  were  instructed  not  to  nominate  themselves,  and  this  created 
undefined  observations  in  «n  otherwise  completely  crossed  experimental 
design.  Gordon  (1969)  and  Willingham  (1959b)  have  provided  methods 
for  estimating  reliabilities  under  these  conditions  but,  unfortu¬ 
nately,  neither  was  totally  appropriate  for  this  data  set  because  some 
nominees  did  not  serve  as  nominators,  as  has  been  previously  mentioned. 

Accordingly,  a  more  general  intraclass  correlation  was  developed  to 
meet  the  requirements  of  this  data  set.  It  Is  of  the  general  type 
presented  in  Hoyt  (1941). 


rkk 


MSg  -  MSe 

M5b 


where: 


rkk  =  tlie  *ntrac*ass  correlation  corresponding  to  k  raters 
MSg  =  the  mean  square  between  nominees'  uncorrelated  mean 
scores-* 

MSe  =  the  mean  square  error 

The  formula  derived  was  then  used  within  each  flight  to  compute  the 
interrater  reliability  for  each  of  the  eight  rating  categories.  The 
resulting  reliability  estimators,  r ^  values,  were  not  directly 
comparable  across  flights  with  differing  numbers  of  nominators  because 


^Uncorrected  scores  could  be  used  at  this  point  because  all 
analyst  were  carried  out  within  flight,  thereby  avoiding  the  problem 
of  non-equiva lent  scores  across  flights. 
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the  reliabilility  to  be  accepted  increases  with  the  number  of 
nominators  (i.e.,  flights  with  50  nominators  would  be  expected  to 
provide  more  reliable  ratings  than  flights  with  25  nominators,  just  as 
a  test  of  50  items  is  expected  to  be  more  reliable  than  one  with  25 
items).  Thus,  to  provide  estimators  which  could  be  considered 
comparable,  the  Spearman-Brown  prophecy  formula  was  applied  to  adjust 
each  estimator  to  a  common  number  of  nominators  as  follows: 


*"37,  37 


37 

=  F  nek 


rkk 


where: 

k  3  harmonic  mean  of  the  number  of  nominators  nomina+ino  each 
nominee  }n  the  flight 

This  coefficient  reflects  the  level  of  interrater  reliability  adjusted 
to  37  nominators  which  is  approximately  the  mean  number  of  nominators 
per  flight.  This  adjustment  permits  fair  comparisons  of  reliabilities 
across  flights,  and  the  distributions  of  these  interrater  agreement 
indices  were  examined  for  each  rating  category  separately. 

In  order  to  describe  the  norming  and  standardizing  of  nomination 
scores  and  th '  derivation  of  a  general  interrater  reliability  formula 
used  in  the  research,  first  configuration  of  data  and  some  terms  must 
be  defined. 

Description  of  lata 

Consider  Figure  1,  the  layout  of  data  for  one  of  eight  dimensions  in 
any  one  of  the  596  flights  of  trainees.  Recall  that  nominators 
attending  a  rating  session  were  allowed  to  nominate  anyone  in  their 
flight  (except  for  them. elves),  including  those  not  present  at  the 
session.  Therefore,  rati>o  data  from  each  flight  involved  M  nomin¬ 
ators  and  N  nominees,  where  •'<  ^M. 

Again  referring  to  Figure  1,  for  each  of  the  NM  combinations  of 
nominators  and  nominees,  a  value,  v^.  (i  *  1,  2,  .  .  .  N;  j  *  1, 
2,  .  .  .  M),  was  assigned,  except  where  the  nominee  and  nominator 
coincided--this  case  is  undefinable  since  nominators  were  Instructed 
not  to  nominate  themselves.  Thus,  there  were  M(N-l)  values  assigned 
according  to  one  of  three  scoring  rules: 


ML.  ’ 


1.  Positive  nominations  only: 


v-f j  *  1  if  the  i**1  nominee  was  one  of  the  five  positive 
nominations  made  by  the  jth  nominator; 

v^  =  undefinable; 

v^j  =  0  if  the  ith  nominee  was  not  one  of  the  five 

positive  nominations  made  by  ty<e  jth  nominator. 

2.  Negative  nominations  only: 

vi j  =  -1  if  the  1th  nominee  was  one  of  the  five  negative 
nominations  for  the  jth  nominator; 

vi i  =  undefinable; 

vi j  =  0  if  the  1th  nominee  was  not  one  of  the  five 
negative  nominations  for  the  nominator. 

3.  Positive  plus  negative  (algebraic  sum)  nominations: 
vij  -  1  if  positively  nominated; 

vi j  *  -1  if  negatively  nominated; 

vi  -j  a  undefinable; 

vij  3  0  if  not  nominated. 

Initially,  th,}  nominations  were  scored  all  three  ways  for  examination 
of  reliability. 

Raw  scores,  X  -j ,  were  fcnen  computed  as  the  arithmetic  mean  of 
available  scores,  Vj  j,  for  the  ith  nominee,  i.e.. 


where: 


M 

Xi  a  2  Vij  Z 
j-1 
(i/j) 


L  =  M-l  for  all  i  <  M  and 


(1) 


Nominators 


1 

2  ... 

M 

Raw 

Scores 

1 

* 

V12  ... 

VlM 

*1 

2 

• 

V21 

• 

* 

• 

V2M 

• 

*2 

• 

• 

• 

M 

• 

• 

VH1 

• 

• 

VM2 

• 

• 

★ 

• 

• 

XM 

M+l 

Vm+1,1 

Vm+1,2  . 

Vm+1,M 

Xm+1 

M+2 

• 

Vm+2,1 

•  • 

VM+2,2  . 

• 

VM+2,M 

• 

XM+2 

• 

• 

• 

N 

•  • 

•  • 

Vni 

• 

• 

Vn2  • 

• 

• 

VNM 

• 

• 

Xn 

M1 

«2 

X. 

v i  i  is  th--  value  assigned  to  the  nominee  by  the  ,  . 

J  nominator  (according  to  one  of  three  sets  of  scoring  rules). 


*  indicates  that  values  on  the  diagonal  are  undefinable  because 
nominators  could  not  nominate  themselves. 


Figure  1.  Layout  of  peer  nomination  data  for  one  flight  of  trainees 
on  one  of  eight  dimensions. 


is  the  raw  score  (mean)  for  the  ith  nominee: 


M 

Xi  =  S 

j=l 
( i*j ) 


'ij 


/ 


Z 


where 


Z  =  M-l  if  i  <_M,  and 

Z  =  M  if  M  <  i  <_  N. 

is  the  mean  of  nominations  for  the  nominator 
(ordinarily  a  constant  if  data  are  complete): 

N 

Mi  »  5  V 
jal 
(it'd) 

X  is  the  grand  mean  of  scores: 


Figure  1  (continued).  Layout  of  peer  nomination  data  for  one  flight 
of  trainees  on  one  of  eight  dimensions. 
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For  the  case  in  which  positive  nominations  only  are  scored,  the  range 
of  Xj  is  therefore  0  to  1;  for  negative  nominations  only,  the  range 
is  -1  to  0;  and  for  the  algebraic  sum  nomination  scoring,  the  range  is 
-1  to  1.  Reliability  of  these  nomination  scores  for  each  of  the  three 
scoring  rules  was  computed  within  flights.  The  algebraic  sum  scores 
proved  to  be  most  reliable  and  all  subsequent  adjusting  of  scores  was 
accomplished  for  these  scores  only  (see  Table  1). 


Table  1.  Interrater  Reliabilities  of  Positive,  Negative,  and 
Algebraic  Sum  Nominations* 


_ Scoring  Methods 

Positive  Negative  Algebraic  Sum 


1. 

Try  Hard 

87b 

93 

93 

2. 

Cooperative 

83 

90 

93 

3. 

People  Oriented 

84 

87 

89 

4. 

Calm/Emotionally  Stable 

80 

88 

88 

5. 

Bright/ Intelligent 

87 

91 

93 

6. 

Not  Excitable 

79 

92 

91 

7. 

Physical  Strength/Energy 

92 

90 

93 

8. 

Likely  to  Succeed  in  Air  Force 

90 

93 

94 

*Mean  reliabilities  were  computed  separately  for  the  male 
(N  »  432)  and  female  (N  *  164)  flights,  and  these  means  were  very 
similar.  Therefore,  ony  the  reliabilities  of  the  combined  groups  are 
presented  here. 

^Decimals  have  been  omitted. 


Developing  Standard  Scores  for  Nominees 

The  strategy  for  creating  nominee  scores  that  may  be  legitimately 
compared  across  flights  included  an  empirical  norming  step  along  with 
the  equating  of  expected  variances  for  different  flights. 
Accordingly,  groupings  of  flights  were  formed  with  approximately  the 
same  number  of  nominees  in  each  flight.  Within  each  of  these  group¬ 
ings,  then,  the  level  of  kurtosis  of  the  distributions  of  raw 
nomination  scores  is  very  similar,  and  this  helps  to  create  comparable 
raw  scores  across  flights  within  these  groupings.  However,  because 
the  number  of  nominators  for  flights  within  each  of  the  groupings 
varied  somewhat  more  than  the  number  of  nominees,  and  differential 
numbers  of  nominators  also  affect  the  comparability  of  scores  across 
flights,  these  scores  were  adjusted  to  equate  the  expected  variances 
in  nomination  scores  for  the  different  flights  (again  within  group¬ 
ing),  which  in  turn  aids  in  creating  scores  that  are  comparable  across 
the  flights. 


At  this  point,  the  adjusted  nomination  scores  for  each  grouping  were 
pooled  forming  a  single  frequency  distribution,  and  percentile  scores 
were  assigned.  Finally,  standard  scores  were  assigned  to  each 

percentile  score  based  on  the  corresponding  percentiles  of  a  normal 

distribution.  Details  of  these  procedures  follow. 

Two  assumptions  were  required  to  proceed  with  the  adjustment  of 

nomination  scores:  (a)  the  underlying  distribution  of  "true" 
nomination  scores  in  large  groupings  of  trainees  is  normal,  and 

deviations  from  normality  a-e  artifacts  of  measurement;  and  (b)  the 

reliability  of  nominations  from  any  one  nominator  selected  at  random 
is  the  same  as  the  reliability  of  any  other  nominator.  The  first 
assumption  is  necessary  to  justify  forcing  a  normal  distribution  of 
scores  for  each  grouping  of  trainees  (with  similar  numbers  of 
nominees)  during  the  final  step  of  the  empirical  norming  process.  The 
second  asumption  Is  required  to  make  reasonable  the  adjustments  of 

nomination  scores  using  a  formula  that  equates  expected  variances  In 

the  distributions  of  those  scores  across  flights. 

Thus,  flights  of  trainees  were  aggregrated  Into  10  groupings,  with 
each  grouping  having  a  range  of  three  or  fewer  nominees  across  the 
flights  (e.g.,  30-33  nominees).  Then,  for  any  one  of  the  rating 

categories,  the  following  summary  statistics  of  the  raw  scores  x  were 
computed; 

Mx  *  mean  of  all  raw  scores  on  a  given  rating  category  within 
a  grouping  of  flights 

a2  =  variance  of  raw  scores 
x 

k  *  mean  of  the  averages  (harmonic  means)  of  nominators  for  all 
all  of  the  flights  within  the  grouping. 

Using  these  statistics,  the  variance  of  the  "typical"  nominator's 
scores  could  be  approximated  by  employing  a  formula  given  by  Gulllksen 
(1951,  pp.  73): 


where  the  reliability  value,  nek  is  the  reliability  of  the  raw 
nomination  scores  of  the  "typical  flight"  within  the  grouping,  i.e., 


(3) 


This  is  the  level  of  reliability  adjusted  to  the  average  number  of 
nominators  in  the  flights  within  the  grouping.  The  variance  of  a 
"typical"  nominator's  scores,  given  in  equation  (2)  above,  is 

Ol) 

important  for  adjusting  nomination  raw  scores. 

I 

Now,  again  within  each  grouping  of  flights,  adjusted  raw  scores,  x, 
were  computed  so  as  to  have  a  mean  of  zero  and  an  expected  variance  of 
approximately  (25)?  as  follows: 

x'  *  g(x  -  x.)  (4) 

where  x.  *  mean  nomination  within  flight  (see  Figure  1)  and 

®x 


x  is  In  turn  the  theoretical  standard  deviation  of  the  nomination 
scores  for  flights  within  the  same  number  of  nominators,  k,  within  the 
grouping  of  flights,  i.e., 


o 


X 


2 


k  +  k  (k  -  1)  rn 


(5) 


Equation  (5)  also  comes  from  Gulliksen  (1950).  The  result  of  this 
particular  adjustment  to  each  raw  score  in  a  grouping  of  flights  Is  a 
distribution  of  adjusted  raw  scores,  x',  with  a  range  of  -100  to  +100 
and  a  standard  deviation  of  25.  The  adjusted  scores  were  rounded  to 
the  nearest  Integer  and  percentile  scores  obtained  for  the  frequency 
distribution  of  these  rounded  scores.  Then,  finally,  standard  scores 
were  assigned  to  each  percentile  score  bared  on  the  percentile 
equivalents  of  a  normal  distribution.  For  example,  a  T-score  of  60 
would  be  assigned  to  the  84th  percentile,  where  the  84th  percentile  is 
one  standard  deviation  above  the  mean  in  a  normal  distribution. 


In  this  manner,  nomination  standard  scores  were  developed  which  are 
comparable  across  flights.  This  comparability  rendered  possible 
correlational  analyses  with  the  nominations,  incorporating  data  from 
trainees  in  all  flights. 

Reliability  Estimation  of  Individual  Flights 

Computation  of  interrater  reliability  for  peer  nominations  poses 
special  difficulties  because  of  the  "diagonal  problem"--1.e. , 
nominators  cannot  nominate  themselves.  This  creates  an  undefined 
observation  in  the  otherwise  completely  crossed  design. 

Two  authors  have  previously  offered  formulas  for  computing  reliability 
estimates  under  these  conditions  (Gordon,  1969;  Willingham,  1959b). 
However,  neither  of  these  formulas  was  suitable  for  the  data  collected 
in  this  study  because  neither  was  readily  adaptable  to  the  case  where 
some  nominees  did  not  serve  as  nominators. 


Thus,  it  was  necessary  to  develop  an  appropriate  index  of  interrater 
reliability  for  the  purposes  of  this  study  as  follows: 


r 

kk 


MS  -  MS 
_ B _ E 

ms8 


(1) 


which  is  the  Hoyt  (1941)  Reliability  formula  for  k  number  of 
nominators  given  the  following  definitions  (see  Figure  1): 


and 


N 

k  5  (xi  -  x.)2 
MSb  =  i=l 

N  -  1 


(2) 


M 

2 


5  j*l  /vii  -  Mi  -  Xi  +  x.V 
MSf  -  i-1  MH.DI  I 

-  (N-ir(H-l)^W 


(3) 


and  k  is  the  harmonic  mean  of  the  numbers  of  observations  used  In 
computations  of  the  Xi: 


(4) 


*  N 
M  +  N-M 
M-l  M 

The  rationale  for  the  estimator,  r f^,  follows  from  conventional 
reliability  theory.  The  parameter  of  estimations  is: 


kk  » 


ka2 


(5) 


k°2  +  o| 


where  a?  is  the  variance  of  "true"  scores,  k  is  the  theoretical 
number  of  nominators  used  to  obtain  the  vector  of  observed  scores, 

andCT£  is  the  variance  of  errors. 


First,  MSe  Is  defined  In  Equation  (3)  to  be  an  estimator  of  o|,  i.e.. 


est  (erg)  =  MSe  (6) 

which  loses  M  degrees  of  freedom  because  of  the  undefined  values, 
Vi Under  the  usual  assumption  that  there  is  no  statistical 
Interaction  of  nominators  with  nominees.  In  an  analysis  of  variance 
sense,  the  estimator,  MSe,  may  be  shown  to  be  unbiased.  Failure  of 

this  assumption  to  be  realized  would  lead  to  an  overestimate  of  erg 
and,  consequently,  an  underestimate  of  p^. 

The  estimate  of  observed  variance  In  scores,  MSg,  may  be  shown  to 

consist  of  two  parametric  components 

E(MSB)  *  +  k°2  (7) 

If  the  asumptlon  of  independence  of  errors  is  made.  Specifically,  It 
Is  assumed  that 

Xi  ■  Ti  +  ei  (8) 

where  Xi  Is  an  observed  score  for  nominee  1  and  Ti  Is  the  "true" 

score  value  for  this  nominee  while  ?i  Is  the  independently 

distributed  error  component.  This  means  that  Xi  Is  Independently 
distributed  with  an  expected  value  of  Ti  and  variance  of 

acrp/z  (9) 

ei  e ' 

where  Z  *  M-l  if  1i  H  and  Z  *  M  If  1  >  H.  The  reader  may  recognize 
Equation  (9)  as  the  variance  of  the  mean  error  across  the  nominators 
for  the  I**1  nominee  by  the  central  limit  theorem. 

Finally,  It  can  be  seen  that  the  numerator  of  Equation  (1)  has  an 

expectation  equal  to  the  numerator  of  Equation  (5)  and,  also,  the 
denominator  of  Equation  (1)  has  an  expectation  which  Is  equal  to  the 
denominator  of  equation  (5),  I.e., 


E(MSb  -  MSe)  =  ka2 

(10) 

E(MSb)  =  k<>2  +  °2 

(ID 

which  are  the  conventional  conditions  of  reliability  estimation. 
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Thus,  Equation  (1),  with  its  elements  defined  in  Equations  (2)  and 
(3),  provided  the  general  intraclass  estimator  for  assessing  the 
reliability  of  nominations  in  individual  flights.  Next,  reliability 
information  was  summarized  to  depict  the  stability  of  the  nominations 
across  the  many  flights. 


Reliability  Estimation  for  All  Flights 

Reliabilities  were  computed  within  each  flight  and  for  each  rating 
category  using  Equation  (1).  Then,  each  of  these  reliabilities  was 
adjusted  to  provide  estimates  that  (a)  yield  fair  comparisons  of  the 
degree  of  reliability  for  different  sized  flights,  and  (b)  give  a 
realistic  picture  of  the  reliability  levels  to  be  expected  in  future 
peer  evaluation  research  with  Air  Force  basic  trainees. 

The  rationale  for  the  adjustments  is  that  the  magnitude  of  reliability 
depends  partially  on  the  number  of  nominators  providing  ratings.  The 
larger  the  number  of  nominators,  the  greater  the  expected  level  of 
interrater  reliability,  just  as  the  longer  form  of  a  test  typically 
provides  higher  reliabilities  than  a  shorter  form.  Importantly, 
increments  in  reliability  in  both  of  these  cases  can  be  closely 
estimated  by  the  Spearman-Brown  formula,  and  we  used  this  principle  to 
"correct"  reliabilities  (up  or  down)  to  the  37  nominator  level  using 
the  formula: 


r  kk  (12) 

r37,  37  TV  T\ 

'  ♦  (r  -  V  rkk 

where  k  =  number  of  nominators  in  the  flight 

r^  =  reliability  of  the  nominations  in  this  flight 

r37,  37  =  the  reliability  adjusted  to  the  37-nominator  level. 

The  37-nominator  level  was  used  because  it  is  approximately  the  mean 
number  of  nominators  in  the  596  flights  studied,  and  it  is  assumed 
that  future  peer  evaluation  research  with  basic  trainees  will  involve 
groups  of  approximately  the  same  size.  Therefore,  these  37-nominator 
reliabilities  should  provide  good  estimates  of  the  level  of 
reliability  that  mght  be  expected  in  future  research. 
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III.  RESULTS  . 


Interrater  Reliability 

Reliabilities  tor  the  positive,  negative,  and  algebraic  sum^  of  the 
positive  and  negative  nominations  were  computed  for  each  flight 
separately.  To  aid  in  selecting  a  scoring  paradigm  for  the 
nominations,  the  means  of  these  reliabilities  were  then  compared. 
Rationale  for  using  mean  reliabilities  was  as  follows:  First,  the  use 
of  means  was  motivated  by  the  practical  need  to  characterize  the 
reliabilities  with  single  summary  indices.  And  second,  it  was 
anticipated  that  differences  between  this  kind  of  estimate  of  "average 
reliability"  and  other  estimates  that  might  be  developed  would  be  very 
slight  because  of  the  "tight"  distributions  experienced  (see  Table  2 
for  a  depiction  of  these  distributions). 

Though  it  may  be  possible  to  develop  a  single  estimator  within  an 
analysis  of  variance  framework,  it  was  considered  impractical  because 
of  the  unnecessarily  complex  statistical  development  which  would  be 
required  to  weight  appropriately  the  variance  components  of  the 
individual  flights  containing  differing  numbers  of  nominators. 
Moreover,  the  computer  programming  for  such  an  estimation  process 
would  have  been  considerable.  Accordingly,  the  mean  of  the  Intraclass 
estimators  for  a  given  dimension  (which  adjusted  using  the 
Spearmen-Brown  prophecy  formula  to  a  common  frame  of  reference--37 
nominators)  was  accepted  as  a  reasonable  estimate  of  the  reliability 
of  a  randomly  selected  flight  with  37  nominators. 

Table  1  clearly  shows  that  the  algebraic  sim  scores  are  most  reliable, 
although  differences  between  these  reliabilities  and  those  of  the 
negative  nomination  scores  are  small.  This  slight  superiority  In  the 
stability  of  the  algebraic  sum  scores  suggests  an  advantage  of  using 
these  scores  in  subsequent  research.  In  addition,  the  summed  scores 
contain  more  information  than  either  the  positive  or  negative  scores 
alone,  and  this  feature  of  the  summed  scores  also  argues  for  their  use 
in  future  research. 

Table  2  provides  a  more  complete  picture  of  the  reliability  results 
for  the  summed  scores.  It  showed  that  reliabilities  within  flights 
are  uniformly  high,  and  therefore,  that  very  stable  peer  nomination 
data  can  be  expected  for  flights  of  this  size  (i.e.,  Ns  «  37). 


4Raw  Scores  for  the  algebraic  sum  scoring  procedure  were 
computed  by  simply  adding  together  all  positive  nominations  and 
subtracting  the  sum  of  the  negative  nominations. 


Norm  tables  were  developed  to  score  these  nominations  or  nominations 
gathered  in  future  peer  evaluation  research.  The  tables  permit  the 
researcher  to  compute  a  standard  score  for  any  raw  nomination  score. 
The  tables  have  been  prepared  for  many  different  sized  flights,  and 
they  are  presented  in  Appendixes  A  to  H. 


IV.  SUMMARY  AND  CONCLUSIONS 

The  Air  Force  gathered  peer  nominations  on  over  27,000  basic 
trainees.  Interrater  reliabilities  o*  the  positive,  negative,  and 
algebraic  sum  nominations  were  computed  within  flight,  and  the 
distributions  of  these  reliabilities  were  examined.  Reliability  of 
the  algebraic  sum  nominations  proved  to  be  highest;  means  of  the 
intraclass  correlations  used  to  index  reliability  ranged  from  .88  to 
.94  for  the  eight  rating  categories,  indicating  good  stability  for  the 
summed  nominations. 

A  specia1  scoring  strategy  for  the  summed  nominations  was  then 
developed  to  render  comparable  raw  nomination  scores  across  the 
different  flights.  The  standardized  nomination  scores  on  each  of  the 
eight  rating  categories  were  subsequently  correlated  with  test  scores 
available  for  many  of  the  trainees.  The  patterns  of  these 
correlations  provided  some  evidence  of  convergent  and  discriminant 
validity  for  the  nominations. 

Conclusions  based  on  this  research  are: 

1.  In  future  peer  evaluation  studies,  the  Air  Force  should  continue 
to  gather  both  positive  and  negative  nominations.  The  algebraic 
sum  of  these  nominations  appears  to  offer  the  most  reliable 
measure  of  the  rating  categories. 

2.  The  strategy  developed  for  standardized  scoring  of  nominations 
should  be  used  in  future  peer  evaluation  research.  TV  norm 
table  developed  in  this  research  will  greatly  facilitate  the 
adjusting  of  raw  nomination  scores.  The  capability  of  correcting 
nomination  scores  to  achieve  comparability  in  such  scores  across 
flights,  along  with  the  high  reliabilities  of  these  nominations, 
further  substantiates  the  utility  of  collecting  peer  ratings. 
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APPENDIX  A.  NORM  TABLES  FOR  DERIVING  STANDARDIZED 
PEER  NOMINATION  SCORES  AND  NORM  TABLES  FOR 
DIMENSION  A  -  TRY  HARD 


These  appendices  A  to  H  provide  instructions  and  tables  for  scoring 
peer  nominations  on  each  of  the  eight  rating  categories  used  in  this 
research.  The  scoring  procedures  for  the  first  category.  Try  Hard, 
are  described  in  Appendix  A;  the  same  procedures  can  then  be  used  to 
score  nominations  on  the  other  rating  categories. 

First,  the  rating  instructions  require  trainees  in  each  flight  to 
identify  the  five  best  or  most  effective  trainees  in  their  flight  on 
each  rating  category.  They  are  also  instructed  to  identify  the  five 
worst  or  least  effective  trainees  in  their  flight  on  each  of  these 
categories.  Now,  a  given  trainee's  raw  score  (X)  for  a  category  Is 
computed  using  the  formula: 

X  _  Number  of  positive  nomination  -  Number  of  negative  nominations 
Number  of  nominations  possible 

Because  trainees  are  prohibited  from  nominating  themselves,  the  number 
of  noninations  possible  is  one  less  than  the  total  number  of 
nominators  in  the  flight,  unless  the  trainee  whose  nominations  are 
being  scored  does  not  serve  as  a  nominator  (e.g.,  he/she  is  absent 
from  the  rating  session).  In  that  case,  the  number  of  nominations 
possible  is  equal  to  the  number  of  nominators  in  the  flight. 

After  the  raw  score  for  a  nominee  has  been  computed  for  the  rating 
category,  an  adjustment  must  be  made  in  this  score  to  corrrect  for  the 
effects  on  the  raw  scores  of  differing  numbers  of  nominators  and 
nominees.  This  can  be  accomplished  by  entering  Table  A-l  with  the 
number  of  nominators  and  nominees  in  the  trainees'  flight  and  then 
employing  the  obtained  value  (Q)  in  the  equation 

Y  =  Q  X 

where  Y  is  the  transformed  raw  score* 


*ln  General,  these  scores  will  range  from  -100  to  +100.  It  is 
mathematically  possible  for  them  to  take  on  values  greater  than  +100 
or  less  than  -100  but  this  should  occur  infrequently.  Also,  we  should 
mention  that  the  transformation  indices  in  Tables  A-l,  B-l,  etc.,  are 
in  the  main  based  on  large  norming  groups  and  therefore  are  probably 
stable  estimates  that  would  be  confirmed  in  future  peer  rating 
research  with  these  rating  categories  and  the  same  kind  of  basic 
trainee  sample.  However,  as  the  Ns  in  these  tables  indicate,  the 
values  for  the  smallest  group  sizes  (e.g.,  N  =  29)  and  the  values  for 
the  largest  group  sizes  {e.g.,  N  =  58)  are  based  on  smaller  samples, 
and  therefore,  the  adjustments  to  raw  scores  made  for  these  groups 
should  be  performed  with  caution.  m 
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The  standardized  score  for  the  trainees  on  this  rating  category  can 
now  be  obtained  by  entering  Table  A-2  with  the  transformed  raw  score 
and  the  number  of  nominees  in  the  trainee's  flight.  An  example  is  now 
provided  to  show  the  scoring  procedure: 

Number  of  nominators  =  37  (the  trainee  did  provide  nominations) 
Number  of  nominees  -  40 

Number  of  positive  nominations  for  the  trainee  on  Category  A  *  14 
v  -  14  -  4 
36 

X  =  +  .278 

The  Q  value  from  Table  A-l  is  92.534 
Thus,  Y  *  +.278  (92.534) 

Y  =  +25.72  or  +26 

and  the  standardized  score  (from  Table  A-2)  =  62 

The  same  procedures  should  be  followed  in  scoring  nominations  on  other 
rating  categories.  The  requisite  tables  are  included  in  Appendixes  B 
to  H. 

As  mentioned  in  the  text  of  the  report,  this  standardizing  process 
yields  scores  that  are  comparable  across  flights  with  differing 
numbers  of  nominators  and  nominees.  Accordingly,  the  standardized 
scores  permit  analyses  of  data  from  multiple  flights,  a  critical 
requirement  for  conducting  further  research  that  focuses  on 
correlations  between  peer  nominations  and  criterion  variables. 


I  K  N  O  ^CWOMN  O'  u\a  N  &  cr  V  *4  <T 
«  I  W  0»O  •  4  (TN  WW  9'«0f4^)oninN9«9 

If*  I  9N  tfiVH  n  «fi  •ON  H)tflK0Of4N^>#IA 

II  •  •*•«•#•#•  •••••••«•• 

«  i  (To»ohwhhnw  NNNNinnnnn 

tft  »  *4  M  N  M  N  W  M  «\J  M  M  NNNNNNNNIVN 
j  «4Ht4<H«4HHHHv4 


i  w  ift  *<  ift-i  irw  t0  <r>n  «  k.  •  ro  eu  in  «r 
4  I  ^WO  W«CTO*Ktf>H  so  *4  ^  N.  O'  M  <SJ  M  M  *4 
lAKCmoMM  llAtfl 
II  ••••••••••  •  •«•••••»• 

(\J  |  •»'fU\U"kl/tlMJtN0i0\0  ifi  »D  i|  «0  k  kK  K  K.K 
UN  I  oooooooaao  ooosoaaoOo 
|  »4«4*4<Hrt«4Wf4t4*4  H  •<  «4  *4  «4  »4  *4  «4  H  *4 


H9*Wf09»»4o>PO*fl'  | 

OU*(T®^lAfOON4  H«4CH)rlKN  I 

NN«ff>0*4NW»^  tMft«ONK0«O>  I  « 

•  *•••«••••  •  ••••••«(  its 

WWWW  *  *  *  *  J  *  *  *  * 

nnnnnnnnnn  nnnnnnnn  | 

n*4r4«4H*4«4H«4f4  WHHHt4»4HH  | 


«0  .*  v0  ^<ro  (TUMT'O  (Ttpan  | 

O  H  KK)  «  I 

kK  W  i  ^  IMA  I  I  •  •  | 

•  ••••♦••••  ••••lllll 

kkkk0««000  «  *  «  «0  »  till 

oaooaaeoao  o  o  o  O  I 

«4^f4H»4*4*4f4»4H  «4  *4  vl  *4  I 


23 


NORMATIVE 

SAMPLE  N*  29  129  399  ll»f  2924  59t T  896S  5525  1215 


« 

• 

• 

in 

• 

1  1  I  1  1 

•  1  I  I  1 

•  fill 

1  1  1  1  1 

m 

• 

1 

•  I  1  I  1 

1  1  1  1  1 

•  1  1  I  I 

1  I  •  1  1 

« 

1 

U. 

IA 

1 

o 

»-« 

«* 

• 

U> 

1 

O'  O  O  H 

NNNNN 

NWNW  1 

*  <0  *  I A 

UJ 

1 

N  N  N  N 

<NJ  M  CM  <N1  M 

NNNNN 

NNNNN 

M  O 

<VJ 

1 

UJ  «C 

in 

1 

Of  O. 

n 

o 

1 

U1 

IA 

1 

0  c  (T  O'  a 

aaot4r< 

*4NNNN 

♦0  W  .0  IA  (0 

1 

1 

• 

»4  H  *4  *4  N 

NNNNN 

NNNNN 

NNNNN 

X 

O' 

1 

< 

1 

Of 

(/» 

o 

•  • 

1 

UJ 

UJ  ^ 

1 

•4  NM  W  1 

U\  IA  %0  (0  « 

IMI«4<4N 

N  10  IO  O  * 

c 

1  1 

• 

•4  f4  «4  *4  *4 

♦4  «4  4  i4  H 

•4NNNN 

NNNNN 

fV 

u»  us 

1 

o 

•  1 

1 

u. 

X 

W1 

1 

z 

M  IA 

• 

m 

1  -* 

1 

1  I  1  *  IN 

iO  to  «  «0  O' 

o»  m  m  «4 

NWWW  1 

Of 

z  • 

1 

1  1  1  «4  «4 

»4  *4  *4  »4  *4 

kNNNN 

NNNNN 

*- 

•  K> 

1 

• 

n  «r 

1 

Of  o 

i 

o  O' 

z 

u.  < 

CM 

1 

X 

«#• 

1 

1  1  1  I  1 

•  1  I  1  I 

•  1  COO' 

»»4NN« 

M 

U.  1 

1 

1  I  I  I  1 

•  •III 

•  |  H  H  *4 

NNNNN 

UJ  * 

I  o 

• 

Of  Of 

O  ^ 

1 

O  k 

O 

U>  • 

Of  O' 

• 

•  |N 

1 

1  1  1  •  1 

1  1  I  1  1 

•  •IN* 

40  O'  O'  o  e 

9  « 

UI  1 

• 

t  1  1  1  1 

•  ••II 

•  •  •  rl  «4 

f4f4r4NN 

5 

1  *■ 

• 

m  z 

ON 

1 

Q  f* 

1 

Z  M 

X 

<  (A 

1  (0 

• 

W  * 

ON 

1 

1  1  1  1  I 

1  I  1  •  1 

1  •  1  I  • 

1  I  1  1  1 

tA  UI 

•  1 

• 

1  1  1  1  1 

llllf 

•  III* 

1  1  I  I  1 

c 

z  i 

1 

U.  M 

IN 

1 

o  o 

UJ  Of 

IN 

1 

-J  o 

M 

1 

1  •  I  I  • 

1  I  I  1  1 

1  I  •  1  • 

m  u. 

1 

1 

1  1  I  1  I 

1  I  I  I  1 

1  I  1  1  1 

1  •  •  1  • 

< 

W 

1 

►* 

IN 

1 

• 

O' 

1 

CM 

CM 

• 

•  1  1  I  1 

1  I  I  1  I 

•  •111 

t  1  f  1  • 

• 

1 

1 

•  till 

1  1  1  1  I 

•  1  1  1  I 

•  III* 

O' 

1 

<NJ 

1 

•  III! 


I  •  I  I  I 
I  I  «  I  I 


f  I  •  I  I 

•  •III 


•  •  <0  m  o»  i 

•  I  NMM  I 


N  «SI  N  CO  M  NNNN« 


O  I 

Ui  UJ  I 

r  or  • 

Of  O  I 
O  O  »  O 

U.  M  I  O  0*  O'  O'  O' 
Ml  |W|||  | 
ZS  I  I 
«  «(  I 
a  OC  I 
h  I 


tA'CMAJ**  D 

O'  O'  O'  O'  O'  O'  «  «0  «0  40  «3  *5  «0  ®  » 

•  I  I  I  I  I  I  I  I  I  •  I  I  I  I 


«0(0  ifltO  K 
MMNMN 


I0KNSK 
M  M  M  M  M 


UU1>  U>  tO  tfl 
NNNNM 


11  IMMA 

NNN  WN 


I*  0-  ^  MM 
NNNNW 


NN  WW  1 

NWWNM 


I  O  H  *4  N 
I  MNNN 


I  I  I  •  I 
I  t  •  I  I 


•  III! 

•  III! 


Q9«  K  tfi 
«  K  K  N.  K 
I  •  I  I  I 


<vj  nnaj  ai 


O'  O*  O' O'* 

INtNNNn 


«««»«  I 


MMMMM  NNNNM 


««»««  I 

« 


J«AtONN 

NNNNN 


UN  «0OKK 
NNNNN 


ununununo 

NNNNN 


N  W  W  W  1 
NNNNN 


•  •III 

•  •III 


I  I  I  •  I 
llllf 


i  n  n  «4 
K  N  K>  N  K 
fill! 


M  N  NN  N 


mm  mm  O' 

NNNNN 


lIKNNN 

NNNNN 


1  UN  UN  (O  K 
NNNNN 


«  •  I  I  I 
•  •  ♦  I  I 


I  O'  O'  O'  • 
•  MMM  M 


O  J 

K  «0  (0  (0  10 

I  I  •  I  I 


NNWWW • 

« 


NNNNN 


NNNNN 


NNNNN « 
I 


I  $  •  (0*  J 
I  '  INN  I 


NNNNN  I 
t 
I 


VA  INN  *4  0 
0  00(00  f 
I  I  I  •  •  I 

I 

I 


TABLE  A-2.  TABLE  OF  STANDARD  SCORES  FOR  TRANSFORMED  RAM-SCORES 
FOR  DIMENSION  A  -  TRY  HARD. 
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table  a-z.  table  of  standard  scores  for  transformed  ram-scores 

FOR  DIMENSION  A  -  TRY  HARO. 
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TABLE  G-l.  T  ABLE  OF  INITIAL  #  RAW-SCORE  TRANSFORMATION  COEFFICIENTS 
FOR  DIMENSION  5  -  PHYSICAL  STRENGTH/ ENERGY* 
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PHYSICAL  STRENGlrl  AND  ENERGY 
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HENSION  G  -  PHYSICAL  STRENGTM/ENEPGY. 
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DEPARTMENT  OF  THE  AIR  FORCE 

AIR  FORCE  HUMAN  RESOURCES  LABORATORY  (AFSC) 
•ROOKS  AIR  FORCE  BASE.  TEXAS  78235 


REFIT  TO 
ATTR  OF:  TSR 


•uwiet,  Removal  of  Export  Control  Statement 


1 6  JAN  1981 


to:  Defense  Technical  Information  Center 
Attn:  DTIC/DDA  (Mrs  Crumbacker) 

Cameron  Station 
Alexandria  VA  22314 

1.  Please  remove  the  Export  Control  Statement  which  erroneously  appears  on 

the  Notice  Page  of  the  reports  listed  H.  This  statement  is 

intended  for  application  to  Statement  B  reports  only. 

2.  Please  direct  any  questions  to  AFHRL/TSR,  AUTOVON  240-3877. 

FOR  THE  COMMANDER 

WENDELL  L.  ANDERSON,  Lt  Col,  USAF  1  Atch 

Chief,  Technical  Services  Oivisior.  List  of  Reports 


Cy  to:  AFHRL/TSE 


